Systems, methods and computer readable medium for visual software development quality assurance

ABSTRACT

A computer-implemented method for identifying discrepancies between a design image of a user interface for an application and a screenshot of the user interface as displayed by the application includes performing a first comparison between the design image and the screenshot to identify one or more discrepancies between the images, excluding from the discrepancies those corresponding to visual elements on the screenshot that include dynamic content, and generating an image of the screenshot, wherein the image includes a visual indication of every discrepancy detected by the second comparison as the identified discrepancies between the design image and the screenshot.

This application claims the benefit of U.S. Provisional PatentApplication No. 63/039,964, filed on Jun. 16, 2020, which isincorporated herein in its entirety.

FIELD OF THE INVENTION

Embodiments of the present invention are directed to softwaredevelopment.

Specifically, embodiments of the present invention are directed tosystems and methods for providing visual quality assurance of softwareapplications under development.

BACKGROUND

User Interface (UI) layout and functionality is an important aspect ofsoftware development. The software development process can include astep in which one or more display screens are generated on a device toprovide a graphical visual interface. These are generated usingexecutable code to provide the front end (whether static or dynamic).The visual elements of the display screen generated by the softwareexecutable can be graphical elements generated for example by a softwaremodule. The visual elements can be images that are retrieved for animage file. Other visual elements can be fonts, colors, icons, menustructures, buttons, etc. The software development process is, in astandard way for development, structured where a designer generatesvisual images (e.g., static images) such as in an image file as thevisual design that the executable software is to deliver when thesoftware is developed and running. A good UI facilitates the use andenjoyment of an application can be dispositive of its success.Significant resources are poured into optimal UI design that effectivelycombine aesthetics and functionality. To ensure that the design isrendered correctly by the application, quality assurance processes areused to compare the design with the UI as rendered by the software.

Quality Assurance (QA) for User Interfaces represents an important stepin the software development cycle. The process involves the comparisonof original designs (i.e. what the designer and the customer agreed on),with the respective UIs (i.e. what the developer implemented in thefront-end and ultimately appears on the screen of a computing device).The QA process is primarily carried out by patient and meticulous QAtesters. Needless to say, it can be tedious. Indeed, visual qualitycheck of the output of the software development process can be timeconsuming, difficult, and imprecise. In addition, the QA process fordesigns is sensitive to human judgement and the outcomes can vary fromindividual to another. Further, the software application often times isdeveloped for multiple platforms, devices, screen sizes (or resolution),or operating environments. For example, one starting display screen (orportion thereof) may have many corresponding application implementations(e.g., IOS, Android, Windows, etc.). Known QA processes are not capableof assisting in software development with tools that address suchcomplex situations or provide related features.

Systems and methods have been proposed to automate or otherwisefacilitate the QA process for UIs. In particular, systems exist thatimplements the ability to automatically compare the original design of aUI with the actual UI as displayed by an application in order toidentify and correct deficiencies in the application's rendering.However, such systems present several deficiencies. For example,existing systems are unable to detect dynamic content in their analysesof UIs. This inability to account for parts of the screen that change(e.g., animation, widgets, etc) generates false positives duringautomatic comparisons between designs and displayed UIs.

What is desired is a system and method for providing quality assuranceof UIs that is more accurate than a human tester and can operate at apixel level. It is also desired to provide a system and method that candetect and account for dynamic content on the screen and minimize thegeneration of false positives.

SUMMARY OF THE INVENTION

Embodiment of the present invention disclose a quality assurance systemfor visual software development. The quality assurance system includes aquality assurance application implemented on a computer using computerreadable software instructions stored in non-transient memory, andconfigured for identifying discrepancies between a design image of auser interface for an application and a screenshot of the user interfaceas displayed by the application, wherein the quality assuranceapplication is further configured to perform computer-implemented stepsincluding: performing a first comparison between the design image andthe screenshot to identify one or more discrepancies between the images;excluding from the discrepancies those corresponding to visual elementson the screenshot that include dynamic content, the excluding including:identifying which of the discrepancies are structural discrepancies;applying a mask to every visual element corresponding to a structuraldiscrepancy on both the design image and the screenshot, wherein themask is shaped like the visual element; performing a second comparisonbetween the masked design image and masked screenshot, wherein a lack ofdiscrepancies detected by the second comparison between a masked visualelement on the design image and the corresponding masked visual elementon the screenshot indicates that the visual element on the screenshotincludes dynamic content; and generating an image of the screenshot,wherein the image includes a visual indication of every discrepancydetected by the second comparison as the identified discrepanciesbetween the design image and the screenshot.

In some embodiments, the steps further include generating and displayinga discrepancy map showing areas of discrepancy between the design imageand the screenshot as shaded areas.

In some embodiments, performing the first or the second comparisonincludes traversing the design image and the screenshot using an SSIManalysis on every pixel.

In some embodiments, the one or more discrepancies include colorpatches, missing elements, and structural discrepancies. In some ofthese embodiments, each of color patches, missing elements, andstructural discrepancies is identified based on different combinationsof local luminance similarity, local contrast similarity, and localstructure similarity.

In some embodiments, applying the mask includes applying acontrast-based mask than applies a shadow to regions of the visualelement corresponding to the structural discrepancy where contrast ishigher than a small value.

In some embodiments, the system further includes generating a bug reportthat includes an inventory of the identified discrepancies, theirlocation, and measures of the divergence of their corresponding visualelements from the design image.

Embodiments of the present invention also disclose acomputer-implemented method for identifying discrepancies between adesign image of a user interface for an application and a screenshot ofthe user interface as displayed by the application. The method includesperforming a first comparison between the design image and thescreenshot to identify one or more discrepancies between the images;excluding from the discrepancies those corresponding to visual elementson the screenshot that include dynamic content, the excluding including:identifying which of the discrepancies are structural discrepancies;applying a mask to every visual element corresponding to a structuraldiscrepancy on both the design image and the screenshot, wherein themask is shaped like the visual element; performing a second comparisonbetween the masked design image and masked screenshot, wherein a lack ofdiscrepancies detected by the second comparison between a masked visualelement on the design image and the corresponding masked visual elementon the screenshot indicates that the visual element on the screenshotincludes dynamic content; and generating an image of the screenshot,wherein the image includes a visual indication of every discrepancydetected by the second comparison as the identified discrepanciesbetween the design image and the screenshot.

In some embodiments, the method further includes generating anddisplaying a discrepancy map showing areas of discrepancy between thedesign image and the screenshot as shaded areas.

In some embodiments, performing the first or the second comparisonincludes traversing the design image and the screenshot using an SSIManalysis on every pixel.

In some embodiments, the one or more discrepancies include colorpatches, missing elements, and structural discrepancies. In some ofthese embodiments, each of color patches, missing elements, andstructural discrepancies is identified based on different combinationsof local luminance similarity, local contrast similarity, and localstructure similarity.

In some embodiments, applying the mask includes applying acontrast-based mask than applies a shadow to regions of the visualelement corresponding to the structural discrepancy where contrast ishigher than a small value.

In some embodiments, the method further includes generating a bug reportthat includes an inventory of the identified discrepancies, theirlocation, and measures of the divergence of their corresponding visualelements from the design image.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features of examples and embodiments in accordance with theprinciples described herein may be more readily understood withreference to the following detailed description taken in conjunctionwith the accompanying drawings, where like reference numerals designatelike structural elements, and in which:

FIG. 1 illustrates a system for providing visual quality control of userinterfaces according to an embodiment of the present invention.

FIG. 2 illustrates a block diagram of an exemplary system forautomatically detecting and identifying discrepancies in a userinterface of a software application, according to an embodiment of thepresent invention.

FIG. 3 illustrates a visual interface of the system of an embodiment ofthe present invention.

FIG. 4 illustrates a discrepancy map as a combination of a structuremap, contrast map, and luminance map according to an embodiment of thepresent invention.

FIG. 5A illustrates a design-UI pair of images featuring dynamic content580 according to an embodiment of the present invention.

FIG. 5B illustrates the masked out design-UI pair according to anembodiment of the present invention.

FIG. 5C illustrates close ups of visual element on each of the designand screenshot images, before and after masking, according to anembodiment of the present invention.

FIG. 5D illustrates before and after masking details of element on eachof the design and UI images, according to an embodiment of the presentinvention.

FIG. 6 illustrates a graph visualization of a test or an assessment ofUI rendering.

Certain examples and embodiments have other features that are one of inaddition to and in lieu of the features illustrated in theabove-referenced figures. These and other features are detailed belowwith reference to the above-referenced figures.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention, provide for a software developmentapplication that receives and loads an image that is a designer'sillustration of a display screen (or a portion thereof), which may havebeen entirely or partially manually prepared by an illustrator on anillustration software application, and receives and loads acorresponding display screen that is generated by the softwareapplication under development (in which the designer illustrated imagesare a specified component of the design for the software applicationunder development).

The display screen can be a screen shot (screen capture) or imagegenerated (e.g., automatically generated) from the software generateddisplay screen. The software development application applies analgorithm in which the software development application traverses thecontents of the image of the designer illustration image and thecontents of the display screen (e.g., in image capture form). Thetraversal can sequential, simultaneously, or some combination thereof.One or more algorithms can be applied as part of the traversal thatidentifies the discrepancies between the illustrated image and thesoftware generated display screen. One or more algorithms can be appliedto identify discrepancies as qualifying as bugs for reporting to thesoftware development team. One or more algorithms can be applied thatautomatically based on the received and identified discrepancies providean adjustment to the software implementation of the display screen toremove or reduce the discrepancy.

A visual interface can be generated that provides a visual illustrationof the discrepancies which can be generated using different parametersfrom the algorithm. The visual interface of generated image can provideintuitive tools for a software developer, design illustrator, or aclient to understand and in response, decide to take action (orautomated action based on the output can be applied to the code).

FIG. 1 illustrates a system 100 for providing visual quality control ofuser interfaces according to an embodiment of the present invention. Thesystem includes a software application 120 installed on an electronicdevice 130 and quality control application 140 implemented on a server150. The electronic device 130 is preferably a desktop computer that cancommunicate with server via mobile networks or other wireless networks.Each of the electronic device and servers is a computer system thatincludes a microprocessor and volatile and non-volatile memory toconfigure the computer system. The computer system 130 also includes anetwork connection interface that allows the computer system 130 tocommunicate with another computer system over a network. Each softwareapplication may also be used independently or in conjunction withanother software application to strengthen or supplement functionalitiesof the other software application.

The quality control application 140 is configured to provide a visualinterface that enables a customer to interact with the system. In someembodiments, the quality control application is a web applicationprovided by the server and configured to run in a conventional browser120. The visual interface may thus be displayed in a conventionalbrowser 120 on the client computing device 130.

FIG. 2 illustrates an example block diagram of a system 200 forautomatically detecting and identifying discrepancies in a userinterface of a software application, according to an embodiment of thepresent invention. The system 200 receives as inputs a pair of images, afirst image 210 representing the original design of a user interface,and a second image 220 being an image of the actual user interface, thatis the application's render of the original design 210 of the userinterface. The original design image 210 or reference image 210 is theimage of the user interface as conceived before or during thedevelopment of the application by graphic artists or developers, and isthe model for the user interface rendered by the application. The designimage 210 includes one or more visual elements 215. The reference image210 may be generated through a variety of means, such as a graphicdesign application, a CAD application, or any other means of generatinga picture of the intended design. The user interface image 220 orapplication render 220 is preferably a screenshot of the application.The user interface 220 may also otherwise have been generated orrendered by the application, provided that it is an exact representationof the user interface 220 as displayed on the screen. Each of originaldesign image 210 or reference image 210 and the actual application'simage or screenshot 220 can be in a variety of image file formats. Theseinclude Windows Bitmap (bmp), Portable Image Formats (pbm, pgm, ppm),Sun Raster (sr, ras), JPEG (jpeg, jpg, jpe), JPEG 2000 (jp2), TIFF files(tiff, tif), Portable Network Graphics (png), among others.

The pair of images 210, 220 is first received and processed by anormalization engine 230. The normalization engine 230 is configured toformat or convert the images 210, 220 into a format that enablescomparison. For example, the normalization engine 230 compares the pixeldimensions of the original design image 210 and the screenshot 220. Ifthe dimensions are not identical, the screenshot 220 is resized to matchthe dimensions of the original design image 210. The normalizationengine 230 may also change the format of an image to match the other orof both images to match to match the preferred image format of thenormalization engine 230. In some embodiments, images are converted intoPNG format. The converted design image 210 and the screenshot 220 arethen sent to the comparison engine 240.

The comparison engine 240 is first configured to locate and detectdiscrepancies between the received image and the application screenshot220. To that end, the comparison engine 240 compares the received designimage 210 and screenshot 220 to generate a discrepancy map 250. FIG. 3illustrates a visual interface 300 of the system of the presentinvention. The visual interface 300 provided by the application enablesthe user to interact with the system. In some embodiments, theapplication my run on a server as a web app, and the visual interface ofthe application may be displayed in a conventional browser on a clientcomputing device (see FIG. 1). The visual interface 300 displays theintended design image 310 and the application screenshot 320 bothpreviously uploaded to the system as described above. Each of the images310 and 320 exhibits visual elements 305. The visual interface furtherdisplays a discrepancy map 330.

The discrepancy map 330 is an image that shows the result comparisonbetween the two images 310, 320. In particular, the discrepancy map 320shows locations 335 where the comparison engine has detected somedifference between the design image 310 and the corresponding screenshot320 of the application in development. Thus, the parts of the imagesthat are identical are blank on the map, and the parts where differencesexist between the reference image 310 and the actual user interface 320are shaded 335. The darkness of the shading 335 in any one locationreflects the severity of the discrepancy at that location.

The comparison engine may generate the discrepancy map 330 using avariety of image comparison techniques. In one embodiment, thediscrepancy map is generated by performing a pixel-wise comparisonbetween original image and the UI image. Specifically, the comparisonengine traverse the content of the images 310, 320 using a StructuralSimilarity Index Method (SSIM) algorithm on every pixel of the images toobtain the discrepancy map 330. SSIM a statistical method tocomprehensively evaluate differences between images and is described inWang, Zhou & Bovik, Alan & Sheikh, Hamid & Simoncelli, Eero. (2004).Image Quality Assessment: From Error Visibility to StructuralSimilarity. Image Processing, IEEE Transactions on. 13. 600-612.10.1109/TIP.2003.819861, which is incorporated herein by reference inits entirety. In this embodiment, the method as implemented employs asliding window over the entire frame to calculate a local SSIM for eachpixel. The SSIM analysis yields three measures of local similarity foreach pixel: luminance similarity, contrast similarity, and structuresimilarity. Each pixel similarity measure can serve as a basis for amap, and the luminance map, contrast map, and structure map can begenerated. In some embodiments, the user may select to display each ofthe maps. The combination or superimposition of all three maps yieldsthe discrepancy map. In other words, the discrepancy is made of pixelseach having three dimensions or similarity measure: luminance, contrast,and structure. FIG. 4 illustrates a discrepancy map 400 as a combinationof a structure map 410, contrast map 420, and luminance map 430.

Referring back to FIG. 3, in some embodiments, user-controlledparameters are provided that affect the detection and mapping ofdiscrepancies 335. For example, the scope parameter 336 determines theresolution of the detector: a sharp scope should be used to highlightdiscrepancies in small elements in the design-UI pair, such ascharacters, small icons. A blurred scope may be employed to highlightdiscrepancies in big elements in the design-UI pair, such as entirewords and paragraphs, logos, buttons, images, etc. Threshold value 337for the map is another parameter that may be provided to the user. Thethreshold value determines the level of similarity between the designimage and the screenshot that will be flagged by the comparison engineas a discrepancy. Threshold thus varies between lenient and strict.Slides or other control means are provided on the user interface toenable the user to change each parameter.

The system uses combinations of the measures of local similarity toidentify three different types of discrepancies from the discrepancymap. A first type of discrepancy is color patches, which are identifiedby areas of extreme structure similarity but low luminance or contrastsimilarity. Missing or spurious elements constitute a second type ofdiscrepancy, which is indicated by high contrast. Any otherdiscrepancies that do not fall satisfy the color patch ormissing/spurious element criteria tend to indicate generic structural orgeometrical discrepancies, which indicate discrepancies in the locationand size of the visual elements of user interfaces.

The comparison engine is configured to compile the location and type ofdiscrepancies between the original image and the application image intoa bug report 260 (see FIG. 2). The bug report 260 provides an inventoryof every visual element from the original design that has changed ordiffers from the intended design. The location of each visual elementmay be recorded using any of a number of location or coordinate systemsthat are known in the art of image processing and analysis. Further, thebugs or visual elements that differ from the design image are visuallyidentified on the application screenshot. In the embodiment illustratedin FIG. 3, visual elements that differ from the those of the designimage are surrounded with boxes 340.

Referring back to FIG. 2, a dynamic content detector 270 next processesthe images in order to account for the dynamic elements in the userinterface. Dynamic content may be a combination of text/images that aredynamically generated by some backend process and cannot be expected tomatch pixel by pixel the text and artworks used by the designer. Unlikestatic visual element, dynamic elements or dynamic content can generatefalse positive in the comparison engine because the dynamic visualelement on a screenshot of the actual UI is likely to differ from thestatic picture of the original design. For example, a dynamic visualelement such as an animation becomes a static picture on a screenshothaving a different visual configuration than the original visual elementon the design image. This discrepancy would generate an erroneous bugreport about the visual element by the comparison engine without thespecialized dynamic content process of the present invention. Thedynamic content detector 270 serves to identify the dynamic visualelements of the user interface.

Dynamic content detection is only applied to the visual elements thathave been identified as structural or geometrical discrepancies. Thedynamic content detector 270 is applied to such elements to determinewhether the dynamic content has caused the mismatch. In particular, thedynamic content detector 270 applies a mask or filter to cover allstructural discrepancy elements in the area under diagnosis (i.e., wherethe suspected mismatch between the original image and the screenshotoccurred). Effectively, this filter replaces the suspect elements inboth the original design image and the screenshot with dark patchesbroadly having the shapes of the elements under review. In particular,the mask applies a shadow to regions of the bug or discrepancy wherecontrast is higher than a small positive value, which are detected aselements. Areas of where no contrast is detected are consideredbackground. The dark patches enable the comparison engine 240 to focuson location and shape/size of the structural discrepancy elements ratherthan the specific text/image content associated with the element. If thecomparison engine 240 does not detect any discrepancy in the area underreview after the elements under review are masked, the comparison engine240 will assume the design-screenshot difference is associated withdynamic content and will not be raised as bug to the user.

FIG. 5A illustrates a design-UI pair of images 510, 520 featuringdynamic content 580. Visual element 530, 540, and 550 differ between theimages. Static element 530 is smaller in the screenshot 510 than in thedesign image 520 and therefore will trigger a bug report when processedby the comparison engine according to the steps described above.Similarly, elements 540 are flagged by the comparison engine becausethey differ between the images as they represent different numbers. Inadditional, elements 550 also trigger the comparison engine because thecontent of the image windows 550 has varied between the design image 510and the actual user interface 520.

Next, the dynamic content detector applies a mask to each of theelements flagged by the comparison engine as discrepancies on bothimages. FIG. 5B illustrates the masked out design-UI pair 510, 520according to an embodiment of the present invention. Each mask 560 isshaped like the element and comprises a shadow or black pixels. FIG. 5Cillustrates close ups of visual element 530 on each of the design andscreenshot images, before and after masking (i.e., covered with mask 560in the bottom images). FIG. 5D similarly illustrates before and aftermasking details of element 540 on each of the design and UI images. Asillustrated, the masks 560 approximate the shapes of the elements andcover them completely.

After masks 560 are applied to both images, the comparison engineprocesses the masked-out images anew to detect discrepancies. The maskedout images are traversed pixel-wise according to the process describedabove and the visual element exhibiting discrepancies are recorded. Ifany visual element previously detected on the initial pair of design-UIimages (before the masking) is still flagged as a discrepancy by thecomparison engine after being masked, that element is identified as areal bug in the actual UI. This is because any element flagged aftermasking exhibit geometric or structural discrepancies that are reflectedin the masks. In other words, the size, shape, orientation, or locationof the mask for that element differ between the design image and therendered image, indicating a structural discrepancy independent of anydynamic content. This is the case for element 530 illustrated in FIG.5C, where the masked element 530, 560 is smaller in the maskedscreenshot than in the masked design image. In contrast, elements thatwere previously detected as discrepancies on the first pass through thecomparison engine but that the second round after masking did not flagare identified as dynamic content. This is the case for elements 540illustrated in FIG. 5D, where the masked element 540, 560 retains thesame structure (size, shape, etc.) on both the design image and thescreenshot image. Dynamic content such as visual element 540 asconfirmed by the comparison engine is excluded from further analysis.Though not described in reference to element 550, the same processdescribed above would yield the exclusion of that element as dynamiccontent as well due to the changing photos exhibited by the screenshotcompared to the design image.

After the identifying and excluding dynamic interface elements (e.g.,540), the system can proceed to the full analysis or diagnostic of theremaining discrepancies. The full analysis stage provides additionalinformation to the user regarding visual elements that exhibitdifferences from the design image. For example, the full analysis canreveal three types of geometrical discrepancies or measures ofdivergence between design and UI can be identified from the discrepancymap using the combination of similarity measures described above:element shift (visual elements misplaced by a few pixels), elementdimensions (when the size of an element is wrong), and element mass (anexpression of discrepancy in the element's pixel count). The discrepancydetection, location, and analysis provided by the discrepancy comparisonengine of the present invention provides an unprecedented level ofprecision in evaluating UI interface errors. In particular, thecomparison engine measures and record divergences of each element thatdiffers between the original and image and the application UI image downto the pixel. The detailed discrepancy information can be compiled inthe bug report, which can be provided to the application developmentteam responsible for the application to enable adjustments thereof.

In some embodiments, the user may select an element identified asexhibiting a discrepancy and in response the visual interface of theapplication may display measures of the divergences of the element fromthe source image (e.g., shift, dimensions, pixel count). For example,the application may indicate that an icon or a bloc of text is shifted150 pixels to the left of its position on the design image and that is27% smaller than the equivalent element on the design image. Theapplication may enable the user to manually correct the divergence ordiscrepancy by selecting and manipulating the element (moving,enlargement, etc.) on the screenshot.

In some embodiments, one or more algorithms can be applied thatautomatically based on the received and identified discrepancies providean adjustment to the software implementation of the display screen toremove or reduce the discrepancy. The visual interface of generatedimage can provide intuitive tools for a software developer, designillustrator, or a client to understand and in response, decide to takeaction (or automated action based on the output can be applied to thecode).

Embodiments of the present invention further disclose methods ofmeasuring aspect of visual performance of UI (such as page load and UIrendering times) as perceived by an end user. FIG. 6 illustrates a graphvisualization 600 of a test or an assessment of UI rendering. The testincludes providing a visual element 610 or an object containing text onthe UI (such as an information bubble), and causing the object to moveacross the screen and assessing generally whether the object behaves asintended, and in particular whether the motion of the element is toofast to be read by a user. For example, as illustrated in FIG. 6, thequality control software causes the object to travel from the top of thescreen and decelerate until stop in the middle of the screen, where theapplication awaits user input to continue the test. Upon indication fromthe user (such as clicking a button provided for this purpose by theapplication), the object 610 accelerates thru the bottom of the screen(the X axis). The quality control application of the present inventionis configured to assess whether the object has behaved as intended.

To assess the behavior of the object 610, first an edge detectionalgorithm is applied record the relative position of the object throughits journey down the screen. Corresponding datapoints for the object arerecorded as it moves, which include the coordinates on the graph (pixelpoints x₁,y₁ to x_(n),y_(n)), associated time stamps for eachcoordinate, and other information such as expected pixel density, screenresolution, and physical screen size. Using this object datapoints andscreen information, the application can calculate the speed andacceleration perceived by an end user viewing a screen of specific sizeand resolution (for example, the application may calculate that a 55 in,4K screen yields an acceleration of 2 cm/s² for a certain object made tomove at a certain speed). In some embodiments, motions of multipleobjects can be measured. In some embodiments, the testing applicationmay be configured to use edge detection and pattern recognition toidentify the performance of transient objects as they appear and leavethe screen.

In general, it should be understood that embodiments apply to systems,methods, and computer readable medium.

Inventions are described herein and are claimed by way of explicit andimplicit disclosure as would be understood by those of ordinary skill inthe art.

The application may be implemented on a system, server, computingdevice, or computer. A system, server, computing device, and computercan be implemented on one or more computer systems and be configured tocommunicate over a network.

The computer system also includes a main memory, such as a random accessmemory (RAM) or other dynamic storage device, coupled to bus for storinginformation and instructions to be executed by a processor of one ormore computers of the computer system. Main memory also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by a processor. Suchinstructions, when stored in non-transitory storage media accessible toprocessor, configure the computer system into a special-purpose machinethat is customized to perform the operations specified in theinstructions and provide or be capable of features and functionalitydescribed herein.

The computer system further includes a read only memory (ROM) or otherstatic storage device coupled to bus for storing static information andinstructions for processor. A storage device, such as a magnetic disk oroptical disk, is provided and coupled to bus for storing information andinstructions.

The computer system may be coupled via bus to a display, such as an LCD,for displaying information to a computer user. An input device,including alphanumeric and other keys, may be coupled to bus forcommunicating information and command selections to processor. Anothertype of user input device is cursor control, such as a mouse, atrackball, touchscreen (e.g., on mobile phones) or cursor direction keysfor communicating direction information and command selections toprocessor and for controlling cursor movement on display. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

The computer system may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system to provide specialized features. According toone embodiment, the techniques herein are performed by the computersystem in response to the processor executing one or more sequences ofone or more instructions contained in main memory. Such instructions maybe read into main memory from another storage medium, such as storagedevice. Execution of the sequences of instructions contained in mainmemory causes the processor to perform the process steps describedherein. In alternative embodiments, hard-wired circuitry may be used inplace of or in combination with software instructions.

The term storage media as used herein refers to any non-transitory mediathat stores data and/or instructions that cause a machine to operationin a specific fashion. Such storage media may comprise non-volatilemedia and/or volatile media. Non-volatile media includes, for example,optical or magnetic disks, such as storage device. Volatile mediaincludes dynamic memory, such as main memory. Common forms of storagemedia include, for example, hard disk, solid state drive, or any othermagnetic data storage medium, a CD-ROM, any other optical data storagemedium, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memorychip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus. Transmission media can also take the form ofacoustic or light waves, such as those generated during radio-wave andinfra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to the processor for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. A hardware bus carries the data tomain memory, from which processor retrieves and executes theinstructions. The instructions received by main memory may optionally bestored on storage device either before or after execution by theprocessor.

The computer system also includes a communication interface coupled tobus. The communication interface provides a two-way data communicationcoupling to a network link that is connected to a local network. Forexample, the communication interface may be an integrated servicesdigital network (ISDN) card, cable modem, satellite modem, or a modem toprovide a data communication connection to a corresponding type oftelephone line. As another example, the communication interface may be alocal area network (LAN) card to provide a data communication connectionto a compatible LAN. Wireless links may also be implemented. In any suchimplementation, the communication interface sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link typically provides data communication through one or morenetworks to other data devices. For instance, network link may provide aconnection through local network to a host computer or to data equipmentoperated by an Internet Service Provider (ISP). ISP in turn providesdata communication services through the worldwide packet datacommunication network now commonly referred to as the “Internet.” Localnetwork and Internet both use electrical, electromagnetic or opticalsignals that carry digital data streams. The signals through the variousnetworks and the signals on network link and through the communicationinterface, which carry the digital data to and from the computer system,are example forms of transmission media.

Hardware and software implementation is also illustratively described orunderstood from the attached Visual QA document and the incorporatedpatent application applications mentioned above.

The computer system can send messages and receive data, includingprogram code, through the network(s), network link and the communicationinterface. In the Internet example, a server might transmit a requestedcode for an application program through Internet, ISP, local network andthe communication interface.

The received code may be executed by the processor as it is received,and/or stored in storage device, or other non-volatile storage for laterexecution.

It should be understood that variations, clarifications, ormodifications are contemplated. Applications of the technology to otherfields are also contemplated.

Exemplary systems, devices, components, and methods are described forillustrative purposes. Further, since numerous modifications and changeswill readily be apparent to those having ordinary skill in the art, itis not desired to limit the invention to the exact constructions asdemonstrated in this disclosure. Accordingly, all suitable modificationsand equivalents may be resorted to falling within the scope of theinvention.

Thus, for example, any sequence(s) and/or temporal order of steps ofvarious processes or methods that are described herein are illustrativeand should not be interpreted as being restrictive. Accordingly, itshould be understood that although steps of various processes or methodsor connections or sequence of operations may be shown and described asbeing in a sequence or temporal order, but they are not necessarilylimited to being carried out in any particular sequence or order. Forexample, the steps in such processes or methods generally may be carriedout in various different sequences and orders, while still fallingwithin the scope of the present invention. Moreover, in somediscussions, it would be evident to those of ordinary skill in the artthat a subsequent action, process, or feature is in response to anearlier action, process, or feature.

It is also implicit and understood that the applications or systemsillustratively described herein provide computer-implementedfunctionality that automatically performs a process or process steps.This can involve human-software interaction such as if desired byselecting images or selecting controls for the process thatautomatically carried out.

It is understood from the above description that the functionality andfeatures of the systems, devices, components, or methods of embodimentsof the present invention include generating and sending signals toaccomplish the actions.

It should be understood that claims that include fewer limitations,broader claims, such as claims without requiring a certain feature orprocess step in the appended claim or in the specification,clarifications to the claim elements, different combinations, andalternative implementations based on the specification, or differentuses, are also contemplated by the embodiments of the present invention.

It should be understood that combinations of described features or stepsare contemplated even if they are not described directly together or notin the same context.

The terms or words that are used herein are directed to those ofordinary skill in the art in this field of technology and the meaning ofthose terms or words will be understood from terminology used in thatfield or can be reasonably interpreted based on the plain Englishmeaning of the words in conjunction with knowledge in this field oftechnology. This includes an understanding of implicit features that forexample may involve multiple possibilities, but to a person of ordinaryskill in the art a reasonable or primary understanding or meaning isunderstood.

It should be understood that the above-described examples are merelyillustrative of some of the many specific examples that represent theprinciples described herein. Clearly, those skilled in the art canreadily devise numerous other arrangements without departing from thescope of the present invention.

What is claimed is:
 1. A quality assurance system for visual softwaredevelopment, comprising: a quality assurance application implemented ona computer using computer readable software instructions stored innon-transient memory, and configured for identifying discrepanciesbetween a design image of a user interface for an application and ascreenshot of the user interface as displayed by the application,wherein the quality assurance application is further configured toperform computer-implemented steps comprising: performing a firstcomparison between the design image and the screenshot to identify oneor more discrepancies between the images; excluding from thediscrepancies those corresponding to visual elements on the screenshotthat include dynamic content, the excluding comprising: identifyingwhich of the discrepancies are structural discrepancies; applying a maskto every visual element corresponding to a structural discrepancy onboth the design image and the screenshot, wherein the mask is shapedlike the visual element; performing a second comparison between themasked design image and masked screenshot, wherein a lack ofdiscrepancies detected by the second comparison between a masked visualelement on the design image and the corresponding masked visual elementon the screenshot indicates that the visual element on the screenshotincludes dynamic content; and generating an image of the screenshot,wherein the image includes a visual indication of every discrepancydetected by the second comparison as the identified discrepanciesbetween the design image and the screenshot.
 2. The system of claim 1,wherein the steps further comprise generating and displaying adiscrepancy map showing areas of discrepancy between the design imageand the screenshot as shaded areas.
 3. The system of claim 1, whereinperforming the first or the second comparison comprises traversing thedesign image and the screenshot using an SSIM analysis on every pixel.4. The system of claim 1, wherein the one or more discrepancies comprisecolor patches, missing elements, and structural discrepancies.
 5. Thesystem of claim 1, wherein each of color patches, missing elements, andstructural discrepancies is identified based on different combinationsof local luminance similarity, local contrast similarity, and localstructure similarity.
 6. The system of claim 1, wherein applying themask comprises applying a contrast-based mask than applies a shadow toregions of the visual element corresponding to the structuraldiscrepancy where contrast is higher than a small value.
 7. The systemof claim 1, further comprising generating a bug report that includes aninventory of the identified discrepancies, their location, and measuresof the divergence of their corresponding visual elements from the designimage.
 8. A computer-implemented method for identifying discrepanciesbetween a design image of a user interface for an application and ascreenshot of the user interface as displayed by the application, themethod comprising: performing a first comparison between the designimage and the screenshot to identify one or more discrepancies betweenthe images; excluding from the discrepancies those corresponding tovisual elements on the screenshot that include dynamic content, theexcluding comprising: identifying which of the discrepancies arestructural discrepancies; applying a mask to every visual elementcorresponding to a structural discrepancy on both the design image andthe screenshot, wherein the mask is shaped like the visual element;performing a second comparison between the masked design image andmasked screenshot, wherein a lack of discrepancies detected by thesecond comparison between a masked visual element on the design imageand the corresponding masked visual element on the screenshot indicatesthat the visual element on the screenshot includes dynamic content; andgenerating an image of the screenshot, wherein the image includes avisual indication of every discrepancy detected by the second comparisonas the identified discrepancies between the design image and thescreenshot.
 9. The method of claim 8, further comprising generating anddisplaying a discrepancy map showing areas of discrepancy between thedesign image and the screenshot as shaded areas.
 10. The method of claim8, wherein performing the first or the second comparison comprisestraversing the design image and the screenshot using an SSIM analysis onevery pixel.
 11. The method of claim 8, wherein the one or morediscrepancies comprise color patches, missing elements, and structuraldiscrepancies.
 12. The method of claim 11, wherein each of colorpatches, missing elements, and structural discrepancies is identifiedbased on different combinations of local luminance similarity, localcontrast similarity, and local structure similarity.
 13. The method ofclaim 8, wherein applying the mask comprises applying a contrast-basedmask than applies a shadow to regions of the visual elementcorresponding to the structural discrepancy where contrast is higherthan a small value.
 14. The method of claim 8, further comprisinggenerating a bug report that includes an inventory of the identifieddiscrepancies, their location, and measures of the divergence of theircorresponding visual elements from the design image.